Method and apparatus for image processing using a template image

ABSTRACT

A digital camera includes a memory card. The memory card is previously recorded with template image data that has been subjected to conversion of Y-data effective range and JPEG compressing. A CPU expands this template image data by a JPEG method, and then subjects only expanded data greater than a predetermined value to effective-range conversion that is reverse to the above. The CPU composites the template image data thus obtained with photographed image data, thereby creating composite image data.

This is a division of Ser. No. 09/096,460, filed Jun. 11, 1998 now U.S.Pat. No. 6,441,850.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a method of compositing a template image datawith a photographed image as well as a digital camera, and moreparticularly to a template/photographed image compositing method anddigital camera arranged to read the template image data that has beencompressed by a JPEG method and previously recorded in a memory card sothat the template image data is read out of the memory card and expandedfor compositing with photographed image data.

This invention also relates to a method and apparatus of recording atemplate image, and more particularly to a template image recordingmethod and apparatus adapted to record template image data into a memorycard through compression.

2. Description of the Related Art

In a conventional digital camera, template image data is previouslyrecorded in a memory card. An operator can create a composite image of atemplate image with a photographed image by desirably selecting atemplate image.

However, the template image data is recorded in the memory card in astate of compression according to a JPEG method. There has been aproblem in that the template image data after expansion contains noiseresulting from errors induced between compression and expansion of thedata, resulting in a problem that noise occurs in the composite image.

SUMMARY OF THE INVENTION

Therefore, it is a primary object of this invention to provide a methodof compositing a template image with a photographed image as well as adigital camera which can improve the image quality of a composite image.

It is another object of this invention to provide a template imagerecording method and a template image recording apparatus which canprevent noise from occurring.

This invention is a method of compositing a template image with aphotographed image, comprising the steps of: (a) preparing a recordingmedium for holding third template image data created by converting firsttemplate image data into second template image data by a firstconverting method and compressing the second template image data by apredetermined compressing method; (b) expanding the third template imagedata by a predetermined expanding method to create fourth template imagedata; (c) converting the fourth template image data having a data valuein a first determined range into fifth template image data by a secondconverting method; and (d) compositing the fifth template image datawith the photographed image data.

The third template image data held in the recording medium is data thatis created by converting first template image data into second templateimage data by a first converting method and compressing the secondtemplate image data by a predetermined compressing method. The thirdtemplate image data reproduced out of the memory medium is expanded by apredetermined expanding method to thereby create the forth templateimage data. The fourth template image data having a data value in afirst determined range only is converted into fifth template image databy a second converting method. The fifth template image data iscomposited with the photographed image data.

In one embodiment of this invention, data having a data value in asecond predetermined range is converted by a first converting methodinto data having a data value in a third predetermined range smallerthan the second predetermined range, while data having a data value inthe third predetermined range is converted by a second converting methodinto data having a data value in the second predetermined range.

In another embodiment of this invention, the second predetermined rangeincludes the first predetermined range, and the first predeterminedrange including the third predetermined range.

In one aspect of this invention, the first template image data includesat least one image-quality-related data component, and the secondtemplate image being created by converting the image-quality-relateddata component by the first converting method. The step (c) includes astep of comparing the image-quality-related data component contained inthe fourth template image data with a predetermined value, and a step ofconverting the fourth template image data including theimage-quality-related data component greater than the predeterminedvalue into the fifth template image data.

According to this invention, the fourth template image data included inthe first predetermined range is converted by a second converting methodinto the fifth template image data. It is therefore possible to removeaway noises due to errors induced between compression and expansion,thereby improving the image quality of a composite image.

This invention is a method of recording a template image, comprising thesteps of: (a) converting first template image data by a predeterminedconverting method to create second template image data; (b) compressingthe second template image data by a predetermined compressing method tocreate third template image data; and (c) recording the third templateimage data into a recording medium.

The second template image data is created by converting the firsttemplate image data by a predetermined converting method, while thethird template image data is created by compressing the second templateimage data by a predetermined compressing method. The third templateimage data thus created is recorded in the memory medium such as amemory card.

According to this invention, the template image data, that has beenconverted by the predetermined converting method and compressed by apredetermined compressing method, is recorded in the memory medium. Thismakes possible to remove noises during reproduction.

This is a digital camera, comprises: a recording medium for holdingthird template image data created by converting first template imagedata into second template image data by a first converting method andcompressing the second template image data by a predeterminedcompressing method; a creating means for creating fourth template imagedata by expanding the third template image data by a predeterminedexpanding method; a converting means for converting the fourth templateimage data having a data value in a first predetermined range into fifthtemplate image data by a second converting method; and a compositingmeans for compositing the fifth template image data with photographedimage data.

The third template image data held in the recording medium is data thatis created by converting first template image data into second templateimage data by a first converting method and compressing the secondtemplate image data by a predetermined compressing method. The thirdtemplate image data reproduced out of the memory medium is expanded by apredetermined expanding method to thereby create the forth templateimage data. The fourth template image data having a data value in afirst determined range only is converted into fifth template image databy a second converting method. The fifth template image data iscomposited with the photographed image data.

According to this invention, the fourth template image data included inthe first predetermined range is converted by a second converting methodinto the fifth template image data. It is therefore possible to removeaway noises due to errors induced between compression and expansion,thereby improving the image quality of a composite image.

This invention is a template image recording apparatus, comprising: acreating means for creating second template image data by convertingtemplate image data by a predetermined converting method; a creatingmeans for creating third template image data by compressing secondtemplate image data by a predetermined compressing method; and arecording means for recording the third template image data into arecording medium.

The second template image data is created by converting the firsttemplate image data by a predetermined converting method, while thethird template image data is created by compressing the second templateimage data by a predetermined compressing method. The third templateimage data thus created is recorded in the memory medium such as amemory card.

According to this invention, the template image data, that has beenconverted by the predetermined converting method and compressed by thepredetermined compressing method, is recorded in the recording medium.Noises can be removed away during reproduction.

The above described objects and other objects, features, aspects andadvantages of the present invention will become more apparent from thefollowing detailed description of the present invention when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing one embodiment of this invention;

FIG. 2(A) is an illustrative view showing one example of a templateimage, FIG. 2(B) an illustrative view showing one example of aphotographed image, and FIG. 2(C) an illustrative view showing oneexample of a composite image;

FIG. 3 is an illustrative view showing a part of operations in the FIG.1 embodiment;

FIG. 4 is a flowchart showing a part of operations in the FIG. 1embodiment;

FIG. 5 is a flowchart showing another part of the operations in the FIG.1 embodiment; and

FIG. 6 is a flowchart showing still another part of the operations inthe FIG. 1 embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring to FIG. 1, a digital camera 10 of this embodiment includes alens 12. An optical image incident upon this lens 12 is given to a CCDimager 14 through a color filter having C_(y), Y_(e,)M_(g) and Garranged in a mosaic form.

When outputting a motion picture through a monitor 34, the CCD imager 14performs so-called well-known pixel mixing readout to supply a resultingpixel signal to a CDS/AGC circuit 16. The CDS/AGC circuit 16 performswell-known noise removal and level adjustment on the inputted pixelsignal. The pixel signal processed by this CDS/AGC circuit 16 is thenconverted by an AID converter 18 into digital data, i.e. pixel data. Afirst signal processing circuit 20 receives the pixel data outputtedfrom the AID converter 18 to calculate luminance data (Y data) andcolor-difference data (U data and V data). Upon creating Y data, theinputted pixel data is averaged according to Equation 1.

For a line (H1+H2)Y={(Mg+Ye)+(G+Cy)}/2={(R+B+R+G)+(G+B+G)}/2=(2R+3G+2B)/2  [Equation 1]

For a line (H3+H4)Y={(G+Ye)+(Mg+Cy)}/2={(G+G+R)+(R+B+B+G)}/2=(2R+3G+2B)/2

where, Mg=R+B, Ye=R+G, and Cy=B+G

Since the Y data is proportional to 2R+3G+2B, the component B isreproduced brighter than the y data (=0.3R+0.59G+0.11B) specified by theNTSC standard. This, however, raises no practical problem.

Meanwhile, when creating U data and V data, the first signal processingcircuit 20 performs subtraction between adjacent pixels according toEquation 2.

For the line (H1+H2)U={(Mg+Ye)−(G+Cy)}={(R+B+R+G)−(G+B+G)}=2R−G  [Equation 2]

For the line (H3+H4)V={(G+Ye)−(Mg+Cy)}={(G+G+R)+(R+B+B+G)}=G−2B

where, Mg=R+B, Ye=R+G, and Cy=B+G

However, the color-difference data is available only every other line.Accordingly, the first signal processing circuit 20 supplements fordeficient color-difference data on a current line by using the colordifference data on the preceding line. That is, V data only is availableon a line (H3+H4) so that the U data on a line (H1+H2) is utilized forthe U data for the line (H3+H4).

The Y, U and V data thus created are written by a memory control circuit26 into a memory area 24 a of a DRAM 24, and are thereafter outputted toa second signal processing circuit 30. The second signal processingcircuit 30, when outputting a motion picture, performs predeterminedhorizontal and vertical interpolations on the Y, U and V data (motionpicture data) from the DRAM 24 so that these data become suited for adisplay-screen size of an LCD 34. The motion picture data, supplied fromthe second signal processing circuit 30, is converted by a D/A converter32 into an analog signal. This analog signal is supplied to the LCD 34,and also outputted through an output terminal 36. As a result, a motionpicture is outputted through the LCD 34.

When an operator depresses the shutter button 40, a system controller 42controls the CCD imager 14 so as to perform so-called all-pixel readout.Due to this, the CCD imager 14 outputs pixel signals at every otherline. Since the CCD imager 14 is mounted with a color filter in a mosaicform, C_(y) and Y_(e) are alternately outputted at an odd line, whileM_(g) and G are alternately outputted at an even line. The CDS/AGCcircuit 16 performs noise removal and level adjustment on the pixelsignal, similarly to the above. The A/D converter 18 converts the pixelsignal from the CDS/AGC circuit 16 into digital data, i.e. pixel data.The CCD imager 14 is disabled after outputting 1 frame of pixel signals.The 1-frame pixel data outputted from the A/D converter 18 is directlydelivered onto a bus 22 without processed by the first signal processingcircuit 20. The pixel data is written into the memory area 24 a by thememory control circuit 26.

The CPU 28 converts the pixel data of the memory area 24 a into Y(=Y.sub.2), U and V data, according to Equation 3 to Equation 5, withusing a working area 24 b. The converted Y, U and V data, i.e.photographed image data, are compressed according to a JPEG format, andwritten into a memory card 46.Y _(h) =C _(y) +Y _(e) +M _(g) +GC _(b)=(C _(y) +M _(g))−(Y _(e) +G)C _(r)=(Y _(e) +M _(g))−(C _(y) +G)  [Equation 3]R=k ₁₁ ×Y _(h) +k ₁₂ ×C _(b) +k ₁₃ ×C _(r)G=k ₂₁ ×Y _(h) +k ₂₂ ×C _(b) +k ₂₃ ×C _(r)B=k ₃₁ ×Y _(h) +k ₃₂ ×C _(b) +k ₃₃ ×C _(r)  [Equation 4]Y _(L)=0.299×R+0.587×G+0.114×BU=B−Y _(L)V=R−Y _(L)  [Equation 5]

Due to the color separation and YUV conversion as stated above, adjacent4 (2 ×2) pixels of C_(y), M_(g), Y_(e) and G data are used to create1-pixel Y, U and V data of the 4 pixels, thereby providing Y, U and Vdata for all the pixels.

The memory control circuit 26 reads Y_(e,)C_(y)M_(g)and G pixel data ona line-by-line basis out of the memory area 24 a, and supplies them tothe second signal processing circuit 30. The second signal processingcircuit 30, when outputting a photographed image, performs colorseparation and YUV-conversion on the pixel data according to Equation 3to Equation 5, thereby outputting a photographed image (freeze image) onthe LCD 34.

If the operator depresses a reproduced button 52, the CPU 28 readsdesired compressed data out of a memory card 46, and expands thecompressed data with using a working area 24 b. The expanded image data(YUV data) is stored in the memory area 24 a, and read out later. Thesecond signal processing circuit 30, in a reproducing mode, performshorizontal and vertical interpolations on the image data. This allows areproduced image to be displayed on the LCD 34.

The memory card 46 may use, for example, a template card that has atemplate image previously prepared as shown in FIG. 2(A) and a templatecode written therewith. With such a card, the operator can set eitherone of a normal reproducing mode or a card-function reproducing mode byoperating a reproducing mode setting button 48.

In the normal reproducing mode, if the operator manipulates a + button54 or − button 56, any one of image data that has been recorded in thememory card 46 is read out. Thus, a reproduced image is displayed on theLCD 34, similarly to the above. That is, if a template image shown inFIG. 2(A) and a photographed image shown in FIG. 2(B) are recorded inthe memory card 46, then either one of the images will be outputtedthrough the LCD 34.

Where the card-function reproducing mode is set, if the operatoroperates the + button 54, the − button 56 and the selecting button 50,an image composited by the template image and the photographed image canbe created as shown in FIG. 2(C).

The template image data shown in FIG. 2(A) is subjected toeffective-range conversion and JPEG compression as shown in FIG. 3, andthen recorded in the template card. That is, 8 bits of the templateimage data are taken into an image processing apparatus (not shown),such as a personal computer, where the Y data constituting the templateimage data is converted in effective range from “0”–“255” to“128”–“255”, according to Equation 6. Incidentally, the range “0”–“255”is defined as a second predetermined range, while the range “128”–“255”as a third predetermined range.d 2=( d1÷2)+128  [Equation 6]

d1: the Y data before conversion on each pixel

d2: the Y data after conversion on each pixel

Consequently, a converted template image is displayed on a display ofthe personal computer. If the operator designates an unwanted portion (aportion other than the curtain), this unwanted portion is cut away. Thatis, this unnecessary portion has a data value turned to “0”. Thepersonal computer compresses the template image data like this accordingto a JPEG format, and records the compressed data into the templatecard, depending upon a designation of recording by the operator.

Incidentally, the compressed data of the template image is accommodatedin an image file with a file name “syn0000S.jpg” (S is an integer),while the compression data of the photographed image is to an image filehaving a file name “pic0000P.jpg” (P is an integer).

The CPU 28 performs the above-stated operation according to a programstored in a flash memory 38. When a card-function reproducing mode isset, the operation depends upon the flowcharts shown in FIG. 4 to FIG.6. In the card-function reproducing mode, the CPU 28 first determines,at a step S1, whether a template code exists in the memory card 46 ornot. If “NO”, the process returns to the normal reproducing mode. Thatis, when a desired memory card is not mounted, even if the operatoroperates the reproducing mode setting button 48, the card-functionreproducing mode cannot be established. If the determination is “YES” atthe step S1, the CPU 28 resets, at a step S3, a count value S of the syncounter 28 a, and determines at a step S5 whether the image file“syn0000S.jpg” exists or not. If “NO”, the process returns to the stepS3, while if “YES”, the process proceeds to steps S7–S11 to expand thecompressed data in the image file to store it into the memory area 24 aof the DRAM 24.

Explaining in further detail, the Y data contained in the compresseddata is first expanded at the step S7, and stored in the memory area 24a. Then, the U data contained in the compressed data is expanded at astep S9, and stored in the memory area 24 a. Subsequently, the V datacontained in the compressed data is expanded at a step S11, and storedin the memory area 24 a. The CPU 28 then reversely converts, at a stepS13, an effective range of the Y data stored in the memory area 24 afrom “128”–“255” into “0”–“255” according to Equation 7, and writes theconverted Y data into the working area 24 b. That is, the CPU 28performs on the Y data a JPEG expansion and reverse conversion in datavalue, as shown in FIG. 3.d3=(d2−128)×2  [Equation 7]

d3: the Y data after reverse conversion on each pixel

The CPU 28 subsequently writes, at a step S15, the Y data and the V dataas they are into the working area 24 b, and reads out the Y, U and Vdata written in the working area 24 b at a step S17. This causes atemplate image having a desired luminance to be outputted through theLCD 34.

The CPU 28 thereafter determines at a step S19 whether a selectingbutton 50 is depressed or not. If “NO”, it is determined at a step S21whether the + button 54 or− button 56 is depressed or not. If neither ofthe + button 54 nor the − button 56 is depressed, the process returnedto the step S19, whereby a same image continues to display on the LCD34. On the other hand, if the + button 54 or − button 56 is depressed,the CPU 28 increments or decrements the count value S of the syn counter28 a according to the button operation at a step S23, and the processreturns to the step S5. This changes over an image to be displayed onthe LCD 34. If the selecting button 53 is depressed by the operator, theCPU proceeds the process from the step S19 to a step S25.

In this manner, the template image data merely expanded is held in thememory area 24 a, and the template image data having Y data reverselyconverted in effective range is held in the working area 24 b.

The CPU 28 resets at the step S25 a count value P of a pic counter 28 b,and then determines at a step S27 whether an image file “pic0000P.jpg”exists in the memory card 46 or not. If the determination here is “NO”,the process returns to the step S25, while if “YES”, the processproceeds to steps S29–S33 to write the photographed image data in theimage file over the working area 24 b. That is, the Y data, U data and Vdata constituting the photographed image data are expanded respectivelyat steps S29–S33, and stored in the working area 24 b. The CPU 28subsequently reads the Y, U and V data out of the working area 24 b at astep S35. This causes the photographed image to be displayed on the LCD34.

The CPU 28 thereafter determines at a step S37 whether or not theselecting button 50 is depressed. If “NO”, it is determined at a stepS39 whether the + button 54 or− button 56 is depressed or not. If thedetermination at the step S39 is “NO”, the process returns to the stepS37. However, if “YES”, the count value P is changed by incrementing ordecrementing at a step S41, and the process returns to the step S27.This changes over the photographed image to be displayed on the LCD 34.

If the operator depresses the selecting button 50, the CPU determines as“YES” at the step S37, and reads, at a step S43, the Y data on any pixelfrom the memory area 24 a. It is determined at a step S45 whether the Ydata has a data value lying in a first predetermined range, i.e.“64”≦Y≦“255” or not. Here, the first predetermined range has its lowerlimit taken smaller than the third predetermined range, because theremay be a case that a deviation occurs in the data value after expansiondue to errors caused between expansion and compression. The lower limitof the first predetermined range is set at “64”, because the value “64”lies intermediate between a minimum value “128” of the thirdpredetermined range and “0”.

If “YES”at the step S45, the CPU 28 at the step S45 reversely convertsthe Y data according to Equation 7 as stated above, and writes thereversely-converted Y data into the working area 24 b. This causes thedata of a relevant pixel of the photographed image data held in theworking area 24 b to be updated by a reversely-converted Y data.Incidentally, the Y data value before the reverse conversion is lessthan “128”, the data value after conversion becomes smaller than “0”. Insuch a case, the CPU 28 forcibly renders the reversely-converted datavalue “0”.

The CPU 28 further writes, at steps S49 and S51, the U data and the Vdata on a same pixel as they are to the working area 24 b. Consequently,the U data and the V data of a relevant pixel of the photographed imagedata had in the working area 24 b are updated by the template-image Udata and V data. The CPU 28 thereafter advances the process to a stepS53. If “NO” at the step S45, the CPU 28 proceeds the process directlyto the step S53 without performing any process. That is, if the value ofthe Y data is not included within the first predetermined range, it isconsidered that the pixel has no data existing thereon or, even ifexisting, it would be due to a noise, thus proceeding the processdirectly to the step S53.

The CPU 28 determines at the step S53 whether all the pixels havecompleted of Y data readout or not. If “NO”, the process returns to thestep S43. However, if “YES”, it is determined that the template imagedata and the photographed image data have completed of compositing.Then, at a step S55, composite image data (YUV data) is read out of theworking area 24 b. This causes a composite image to be outputted on theLCD 34 as shown in FIG. 2(C). The CPU 28 also compresses, at a step S57,this composite image data according to the JPEG format, and records thecompressed data into the memory card 46. Then the process is ended.

According to this embodiment, a template card is prepared which isrecorded with template image data that has Y data having data valueconverted in effective range and compressed according to a JPEG format.During reproducing the template image data from the template card, the Ydata is reversely converted in effective range. Then, the template imagedata, on except for a pixel having a Y data value smaller than apredetermined value, is composited with photographed image data.Therefore, it is possible to prevent noises from occurring on acomposite image and hence improve the image quality of a compositeimage.

Incidentally, the memory card may use various kinds of cards involvingSSFDC (Solid-State Floppy Disc Card). Although this embodiment wasexplained using a complementary-colored filter having Y_(e), C_(y),M_(g) and G arranged in a mosaic form, a primary-colored filter may beemployed that has R, G and B arranged in a mosaic form. Further,although in this embodiment the Y data is converted in effective range,the U data or the V data may be converted in effective range. Whereusing a primary-colored filter, it is possible to convert the effectiverange of any one of the R, G and B data.

Although the present invention has been described and illustrated indetail, it is clearly understood that the same is by way of illustrationand example only and is not to be taken by way of limitation, the spiritand scope of the present invention being limited only by the terms ofthe appended claims.

1. An image compositing method, comprising the steps of: (a) subjectingto an expanding process template image data created by a firstconverting process and a compressing process being performed in thisorder; (b) subjecting to a second converting process template image datacreated by the expanding process in said step (a); and (c) compositingtemplate image data created by the second converting process in saidstep (b) with photographed image data, wherein the first convertingprocess includes a dividing process for dividing a raw data value by afirst numeral, and an adding process for adding a second numeral to adivided value obtained by the dividing process, and the secondconverting process includes a subtracting process for subtracting thesecond numeral from a data value, and a multiplying process formultiplying a subtracted value obtained by the subtracting process bythe first numeral.
 2. An image compositing method according to claim 1,wherein the compressing process and the expanding process are performedin accordance with a JPEG format.
 3. An image compositing methodaccording to claim 1, wherein the template image data includes luminancedata, and the luminance data is subjected to the first convertingprocess.
 4. A digital camera, comprising: an expander for subjecting toan expanding process template image data created by a first convertingprocess and a compressing process being performed in this order; aprocessor for subjecting to a second converting process template imagedata created by the expanding process of said expander; and a compositorfor compositing template image data created by the second convertingprocess of said processor with photographed image data, wherein thefirst converting process is a process of dividing a raw data value by afirst numeral and adding a second numeral to the divided value, and thesecond converting process is a process of subtracting the second numeralfrom a data value and multiplying the subtracted value by the firstnumeral.
 5. A digital camera according to claim 4, wherein said secondconverting process further includes a changing process for changing asubtracted value less than zero to zero.
 6. A digital camera accordingto claim 4, wherein the compressing process and the expanding processare performed in accordance with a JPEG format.
 7. A digital cameraaccording to claim 4, wherein the template image data includes luminancedata which is subjected to the first converting process.
 8. A digitalcamera according to claim 4, wherein said processor includes a comparerfor comparing a value of each pixel data forming the template image datawith a threshold value, and a converting executor for subjecting to thesecond converting process pixel data having a data value which is equalto or more than the threshold value.